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1.  Introduction 

The  World  Wide  Web  (WWW)  and  its  evolving  technology  provide  Air  Force 
users  with  commercial  information  technology  to  enable  users  more  effective  processes 
for  accomplishing  the  mission.  At  the  United  States  Air  Force  Academy  (USAFA),  this 
technology  provides  faculty  and  staff  members  an  effective  mechanism  for  distributing 
course-related  materials  and  other  information  by  posting  it  on  websites  or  homepages. 
Although  faculty  are  quite  prolific  at  publishing  these  resources,  an  intuitive  navigational 
infrastructure  for  finding  them  once  produced  is  lacking.  Without  an  organizational 
schema,  cadets,  faculty  and  staff  spend  unnecessary  time  searching  for  information  and 
learning  multiple  software  interfaces  for  similar  tasks,  time  which  could  be  spent 
gleaning  pertinent  data  from  the  resource  itself. 

Many  instructors  host  personal  web  servers,  and  many  academic  departments  host 
departmental  web  servers  with  departmental  and  course  related  infonnation.  However, 
the  naming  convention  of  these  servers  and  the  filing  system  of  these  web  pages  are 
cryptic.  Therefore,  without  the  specific  address,  information  within  these  pages  is  often 
impossible  to  locate,  even  when  the  instructor  and  course  number  are  known.  In 
addition,  these  resources  could  not  be  found  from  a  link  within  the  department’s  home 
page.  Therefore,  it  seems  that  a  logical  and  intuitive  navigational  interface  is  needed  for 
web-based  resources  at  USAFA.  This  could  be  accomplished  by  generating  web  pages 
for  each  cadet  and  faculty  member,  which  act  as  a  gateway  to  pertinent  and  essential  web 
resources.  This  educational  gateway  should  be  customizable  to  act  as  a  personalized 
information  manager  that  will  place  on  the  desktop  those  items  that  are  accessed 
frequently  or  are  of  high  importance. 

For  example,  the  cadet’s  web  page  would  display  links  to  their  specific  sections  of 
their  academic  courses,  the  uniform  of  the  day,  information  on  military  and  athletic 
training,  and  a  fully  integrated  calendar  with  links  to  assignment  descriptions  and  practice 
exams  (Figure  1).  These  web  pages  could  be  generated  from  databases  prior  to  the  start 
of  each  semester  and  allow  cadets  to  have  textbook,  lesson  objectives  and  reading 
assignment  information  in  advance  of  the  first  lesson  when  syllabi  are  traditionally 
handed  out.  New  information  could  be  added  to  the  cadet’s  personalized  educational 
gateway  system  database  to  keep  them  informed  of  upcoming  changes  to  the  schedule  of 
calls  or  other  happenings  in  the  cadet  wing.  For  faculty,  uniform  information  could  also 
be  provided  as  well  as  links  to  the  course  sections  they  are  teaching,  course  rosters, 
research  databases  and  necessary  infonnation  for  their  additional  duties. 

Another  problem  facing  faculty  and  cadets  at  the  Air  Force  Academy  is  a  lack  of 
web  development  tools  and  guidelines  fully  supported  by  both  the  academic  faculty  (DF) 
and  the  communications  squadron  (SC).  Currently,  the  Center  for  Educational 
Excellence  gives  workshops  on  how  to  use  Microsoft’s  FrontPage,  yet  based  upon 
recommendations  from  the  Air  Force  Communication  Agency  (AFCA),  SC  supports 
Netscape’s  Composer  and  Macromedia’s  Dreamweaver.  Other  faculty  members  use  web 
development  software  that  is  not  supported  by  either  organization.  The  standardization 
of  web  development 


Figure  1:  An  example  of  a  homepage  on  a  cadet’s  desktop  developed  by  the  Cadet 
Personalized  Educational  Gateway  (CPEG)  navigational  interface. 


tools  is  difficult  because  different  software  packages  are  favored  for  the  development  of 
different  applications,  and  there  is  no  one  package  that  seems  sufficient  for  all  skill  levels 
of  developers  (AFC A  TR  99-10). 

However,  a  web  development  software  package  with  templates  specifically 
designed  for  academic  courseware  (e.g.  a  web  course  management  tool)  that  is  supported 
by  both  DF  and  SC  would  be  beneficial  for  faculty  inexperienced  in  web  design  and 
authoring.  Web  course  management  tools  are  specifically  designed  to  incorporate  many 
popular  course  features  such  as  collaborative  white  boards  and  discussion  groups,  thus 
negating  the  need  for  developers  to  learn  multiple  packages.  In  addition,  these  authoring 
systems  facilitate  reusable  course  content  allowing  instructors  to  incorporate  pre-existing 
learning  modules,  reducing  the  duplication  of  effort  to  produce  fundamental  widely  used 
course  content.  This  will  maximize  the  time  an  instructor  has  to  devote  to  the 
development  of  new  learning  modules  and  course  content. 


2.  Purpose 

The  purpose  of  this  paper  is  to  outline  a  comprehensive  and  flexible  web 
navigational  concept,  the  Cadet  Personalized  Educational  Gateway  (CPEG).  To 
understand  how  CPEG  would  interface  resources  developed  by  other  software,  I  have 
provided  background  information  on  web  development  software  commonly  used  in 
education.  This  is  followed  by  a  description  of  the  interface  itself,  a  list  of  CPEG 
requirements  and  a  description  of  the  evaluation  of  the  user  interface.  The  investigation 
team  and  a  timeline  for  the  study  are  included  at  the  end  of  this  document.  This  report 
defines  the  research  project  leading  to  the  creation  of  a  CPEG  at  the  Air  Force  Academy 
during  the  summer  of  2000  and  a  distribution  of  the  research  report  to  other  Air  Force 
Organizations  in  August  2000. 

3.  Background— Commercial  off  the  shelf  web  development  products 

Currently,  there  are  several  hundred  commercial  off  the  shelf  (COTS)  products 
available  for  developing  web  applications  for  educational  use.  The  selection  of  products 
is  highly  volatile  as  new  products  are  added  and  other  products  are  removed  from  the 
market  on  a  daily  basis.  These  programs  range  in  functionality  from  Netscape’s 
Composer,  which  may  be  sufficient  for  the  authoring  text-based  web  pages  to 
comprehensive  course  management  packages  such  as  BlackBoard’s  Courselnfo  and 
Universal  Learning  Technology’s  WebCT.  This  section  outlines  the  major  categories  of 
web  development  software  and  their  educational  applications. 

3.1  Web  authoring  software 

HyperText  Markup  Language  (HTML)  is  simply  a  specification  for  marking  plain 
text  files  with  formatting  codes  that  specify  different  fonts  and  styles.  It  is  used  to 
determine  the  way  text,  images,  and  links  are  shown  in  a  WWW  document  and  is  the 
basis  of  all  web-authoring  software.  HTML  can  be  typed  in  as  a  series  of  format  codes  in 
a  plain  text  editor  such  as  Unix  “vi”  or  Microsoft’s  “Notepad”,  or  also  generated  from 
“What  You  See  is  What  You  Get”  (WYSIWYG)  web  authoring  software.  Often  web 
authoring  software  is  initially  used  to  create  the  web  page,  and  the  resulting  HTML  code 
is  modified  by  the  developer. 

In  January  1999,  AFCA  completed  a  study  of  several  COTS  web  authoring 
software  packages.  While  all  of  the  packages  tested  produced  functional  web  pages, 
NetObjects  was  not  recommended  for  advanced  applications  because  the  HTML  code 
that  the  package  generated  was  difficult  to  modify.  Macromedia  Dreamweaver  was 
determined  to  be  the  best  overall  HTML  web  authoring  tool,  yet  it  lacks  a  user  friendly 
interface  necessary  for  novice  users.  While  Microsoft’s  FrontPage  ’98  easily  meets  the 
demands  of  beginning  and  intennediate  web  page  builders,  some  of  the  features  only 
work  when  viewed  using  Microsoft’s  Internet  Explorer  web  browser.  The  product  also 
requires  installation  of  server  extensions  to  non-NT  servers  to  function  on  those 
platforms.  Therefore,  its  usage  is  only  recommended  for  an  intranet  and  in  settings  where 
all  users  accessing  the  web  pages  are  using  Microsoft’s  Internet  Explorer  web  browser. 


3.2  Interactive  web  development  software 

Interactive  or  dynamic  web  sites  are  more  complicated  to  build  than  are  static  web 
pages  built  with  web  authoring  tools,  but  are  very  useful  for  posting  practice  quizzes  and 
other  assignments,  which  provide  online  feedback  to  the  student  and/or  instructor  about  a 
student’s  performance.  Historically,  online  quizzes  and  interactive  course  materials  have 
been  written  with  JAVA,  JavaScript  or  a  common  gateway  interface  (cgi)  shell  script, 
such  as  Perl5.  Cgi-scripts  write  server-side  programs  that  respond  to  web  requests  by  the 
web  site  user.  Perl5  and  other  cgi-scripts  generally  work  across  platforms  and  are 
supported  by  both  Netscape  and  Microsoft’s  Internet  Explorer  browser,  and  require 
programming  knowledge  to  develop. 

However,  web  development  packages  such  as  Allaire’s  Cold  Fusion,  Microsoft’s 
Visual  Studio,  and  Sun’s  NetDynamics  have  built  in  script  authoring  tools  that  enable  the 
developer  to  build  sophisticated  interactive  web  pages,  with  limited  programming 
knowledge  (Seltzer,  1999).  In  addition,  Macromedia  has  a  variety  of  web  development 
tools  specifically  designed  for  building  interactive  multimedia  web  pages  including 
Macromedia  Director,  Flash  and  Attain  Objects  for  Dreamweaver. 

3.3  Collaboration  tools 

Collaborative  tools  include  bulletin  boards  for  posting  announcements  and  file 
sharing  for  group  assignments.  Students  can  be  assigned  to  a  team  project  and  users  at 
remote  sites  can  then  work  together  on  the  same  file,  html  document,  Microsoft 
PowerPoint  or  other  document.  Electronic  whiteboard  features  are  another  collaborative 
tool,  which  can  be  used  in  conjunction  with  mini-desktop  cameras  for  desktop  video¬ 
teleconferencing. 

Discussion  options  include  asynchronous  discussions  where  participants  don’t 
need  to  communicate  at  the  same  time  (threaded  discussions)  and  synchronous 
discussions  where  participants  are  logged  into  a  virtual  area  and  communicate  at  the  same 
time  (“chat”  sessions).  Although  synchronous  discussions  are  rarely  useful  when  more 
than  3-5  students  are  logged  simultaneously  (Gray,  1998),  some  programs  have  session 
log  features  that  store  a  text  of  the  discussion,  thus  increasing  their  usefulness. 

There  are  currently  over  103  software  packages  available  that  have  some  sort  of 
collaborative  tool  feature  -  usually  asynchronous  discussion.  Among  the  more 
comprehensive  packages  that  bundle  together  many  of  these  tools  into  one  package 
include  Microsoft’s  Netmeeting,  GTE’s  InfoWorkspace,  and  CommonSpace’s 
Collaborative  Writing  Software. 

3.4  Course  management  software 

Course  management  software  incorporates  many  different  web  development  tools 
into  a  seamless  interface  for  a  single  course.  Therefore,  faculty  learn  one  software 
package  for  web-based  curriculum,  collaborative  work,  and  multimedia/interactive 
resource  development.  Although  some  faculty  members  are  intrinsically  interested  in 
learning  different  software  for  different  applications,  other  faculty,  particularly  those  less 


computer  literate  or  who  have  demanding  schedules,  will  also  be  able  to  develop 
resources  for  their  courses  more  efficiently. 

Most  course  management  software  provides  built-in  student  tracking  and 
assessment  features.  The  assessment  features  enable  instructors  to  create  and  administer 
tests,  quizzes  and  other  forms  of  assessment  online.  Students  can  get  immediate 
feedback  on  their  performance  along  with  post-test  statistical  analyses.  Using  these  tools, 
instructors  can  put  a  time  limit  on  quizzes,  and  some  of  these  tools  allow  the  random 
generation  of  online  tests  so  that  no  two  students  get  the  same  question  set.  Instructors 
can  weight  certain  questions,  and  some  products  customize  content  provided  to  an 
individual  student  based  upon  that  particular  student’s  performance.  Other  tools  provide 
for  automatic  grading  of  these  exercises  and  the  resulting  gradebooks  are  exportable  into 
Microsoft’s  Excel  or  other  institution  specific  administrative  systems  (e.g.  Grader). 

Student  tracking  features  allow  instructors  to  examine  the  student’s  use  of  these 
online  resources.  Some  products  will  identify  the  IP  address  from  where  the  student 
accessed  the  information  and  provide  detailed  student  progress  information  to  the 
instructor,  including  the  date  and  time  students  accessed  the  information,  and  the  number 
and  duration  of  hits  on  each  course  page.  This  information  can  be  used  to  make 
inferences  about  the  interest  and  difficulty  of  the  content  as  well  as  the  students’  level  of 
effort. 
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Figure  2:  The  CPEG  interface  concept 


4.  CPEG— Description 

The  CPEG  interface  would  build  upon  the  resources  already  available  in  the 
course  management  software  and  other  databases  to  provide  links  for  cadets  and  faculty 
to  online  resources  at  USAFA.  For  example,  the  courses  in  which  a  cadet  is  enrolled 
could  be  accessed  from  a  database  provided  by  the  registrar’s  office.  Instructors  could 
input  homework  assignment  deadlines  into  a  course  management  database  that  appear  on 
a  modifiable  calendar  on  the  cadet’s  web  page.  Cadets  could  post  assignments  to  a  web 
page  automatically  linked  to  the  instructor’s  desktop.  Announcements,  such  as  a  change 
in  the  uniform  of  the  day,  would  be  available  from  changes  made  to  yet  another  database. 
And  changes  in  these  databases  would  automatically  update  the  cadet  and  faculty 
member’s  web  pages  without  rewriting  the  HTML  code.  In  addition  there  would  be  links 
to  the  bookstore,  athletics  and  web  pages  external  to  USAFA  (Figure  2). 

4.1  Requirements 

The  software  used  to  produce  the  Cadet  Personalized  Educational  Gateway 
(CPEG)  and  the  resulting  navigational  interface  should  meet  these  requirements: 

a.  Intuitive  graphic-based  interface 

The  main  purpose  of  this  system  is  to  provide  a  mechanism  for  managing  web 
resources  efficiently  and  effectively.  Icons  and  color  schemes  should  be  carefully  used 
and  the  number  of  steps  (mouse  clicks  and  keystrokes)  needed  to  perform  a  task  should 
be  kept  to  a  minimum. 

b.  Consistent  addressing  scheme 

Web  pages  for  courses  should  use  the  department  and  course  number  for  the 
address  to  facilitate  navigation  without  using  a  search  option.  For  example  Biology  215 
would  be  www.usafa.af.mil/dfb/dfb215. 

c.  Customizable 

While  all  cadets  and  faculty  members  will  have  similar  organizational  needs,  not 
every  individual  will  want  the  exact  same  set  of  resources  on  their  desktop.  For  example, 
some  faculty  may  want  to  link  to  the  Early  Bird,  while  others  may  be  more  interested  in 
the  headlines  from  this  morning’s  New  York  Times.  Therefore  faculty  and  cadets  should 
be  able  to  modify  their  desktop  to  delete  superfluous  links  or  to  add  links  of  interest. 

d.  Password  protection 

Different  levels  of  password  protection  should  be  built  into  the  system.  Faculty 
should  have  access  to  different  areas  and  databases  than  cadets,  and  administrators. 
Similarly  cadets  and  administrators  should  have  privileges  to  some  areas  not  accessible 
by  faculty. 

e.  Calendar  function 

While  all  cadets  and  faculty  members  will  have  similar  organizational  needs,  not 
every  individual  will  want  the  exact  same  set  of  resources  on  their  desktop.  For  example, 
some  faculty  may  want  to  link  to  the  Early  Bird,  while  others  may  be  more  interested  in 


the  headlines  from  this  morning’s  New  York  Times.  Therefore  faculty  and  cadets  should 
be  able  to  modify  their  desktop  to  delete  superfluous  links  or  to  add  links  of  interest. 

f.  Database  generated 

To  keep  maintenance  of  the  system  to  a  minimum,  web  pages  should  be  generated 
and  updated  from  databases.  The  system  should  have  the  ability  to  update 
announcements  and  news  items  on  a  continual  basis. 

g.  Interface  with  other  USAFA  databases 

In  order  for  this  system  to  be  an  effective  gateway  to  internet  resources  at 
USAFA,  it  should  interface  seamlessly  with  the  databases  and  web  interfaces  being 
constructed  elsewhere  on  base  (e.g.,  course  registration  databases). 

h.  Incorporate  functions  that  cadets /faculty  already  do 

Faculty  already  write  syllabi,  develop  worksheets  and  homework  assignments, 
and  enter  grades  and  military  performance  ratings  into  databases.  This  system  should  not 
duplicate  this  effort  and  require  an  instructor  to  perform  a  task  twice. 

i.  Interface  with  other  web  development  software 

This  system  should  provide  a  seamless  interface  with  web  course  management 
software  and  other  tools  that  are  provided  by  USAFA  for  web  development. 

j .  Interface  with  in-house  developed  systems 

This  system  should  not  prevent  faculty  from  pursuing  the  creative  endeavor  of 
producing  new  web-based  applications.  CPEG  and  any  templates  included  with  CPEG, 
should  not  hinder  web  based  educational  applications  being  developed  (for  example)  in 
foreign  languages  and  physics.  To  this  extent,  it  should  support  foreign  language 
character  fonts  and  be  able  to  interface  with  complex  mathematical  equations. 

k.  Relatively  easy  to  learn 

Approximately  23%  of  faculty  and  a  25%  of  cadets  are  new  to  USAFA  every 
academic  year.  Therefore  it  is  critical  that  both  web  based  course  material  development 
and  the  CPEG  interface  be  easy  to  learn  for  novice  users. 

l.  Course  materials  must  be  easy  to  archive 

As  changes  are  made  to  existing  courses  from  one  tenn  to  the  next,  the  option 
should  be  available  to  archive  the  older  versions. 

m.  Reusable  content 

Instructors  should  be  able  to  access  a  departmental  database  of  figures,  quizzes, 
worksheets,  multimedia  resources,  ideas  etc  with  minimal  duplication  of  these  items  on  a 
single  server. 


n.  Collaborative  workspace  and  desktop  video-teleconferencing  capabilities 

Tools  used  to  develop  courses  should  have  group  collaboration  features  including 
presentation  space,  discussion,  collaborative  whiteboards,  and  the  capability  to  adapt 
these  to  future  desktop  video-teleconferencing  projects. 

o.  Meet  IMS  specifications 

Currently,  there  is  a  lack  of  a  standard  set  of  specifications  for  web-based 
resources  and  web  development  tools.  However,  EDUCAUSE’s  Instructional 
Management  Systems  (IMS)  project  is  an  initiative  to  develop  a  set  of  technical  and 
design  specifications,  to  facilitate  growth  and  long-tenn  viability  of  web-based  learning 
resources.  Adopting  these  specifications,  developed  by  members  from  commercial, 
academic,  and  government  organizations,  may  help  ensure  long-tenn  viability  of  the 
project. 

p.  Must  be  able  to  evaluate  prototype  using  existing  IITA  lab  equipment 

Although  the  purchase  of  another  type  of  server  is  not  being  ruled  out,  the  initial 
evaluation  of  the  CPEG  prototype  will  be  performed  on  the  current  hardware  available  in 
the  IITA  lab  (Microsoft  NT  or  Sun  Ultra  5  Unix-based  servers). 

5.  Using  commercial  off  the  shelf  software  to  develop  CPEG 

After  an  exhaustive  search  of  the  literature,  personal  interviews  with  marketing 
representatives  and  faculty  at  other  academic  institutions,  product  review,  and 
demonstrations,  only  one  COTS  product  has  been  determined  to  come  close  to  meeting 
the  requirements  stated  in  Section  4. 1  above.  Blackboard’s  Campus  product  is  comprised 
of  several  modules  including  an  online  personal  information  manager.  This  personal 
information  manager  provides  a  unique  set  of  links  and  announcements  based  on  a  user’s 
identity.  It  presents  students  and  faculty  with  a  calendar  of  upcoming  campus  wide  and 
course  specific  events,  links  to  online  course  resources  and  access  to  other  areas  of  the 
institution’s  intranet. 

Integral  to  this  system  is  a  suite  of  course  management  tools,  which  includes 
student  tracking,  collaborative  workspace,  web  authoring  features  and  a  central  repository 
for  reusable  content.  Therefore,  Blackboard’s  Campus  software  will  be  evaluated  along 
with  other  COTS  software  that  meets  the  requirements  stated  in  4. 1  above  and  becomes 
available  during  the  timeframe  of  this  study. 

5.1  Evaluation  of  the  CPEG  interface 

Very  often  software  purchase  and  implementation  decisions  are  made  by  technical 
personnel  who  base  their  decision  on  personal  use,  attendance  at  vendor  sponsored 
workshops,  reading  about  it  in  trade  publications,  or  having  used  other  products  from  the 
same  vendor.  This  type  of  decision-making  process,  also  known  as  “n  of  1”  does  not  take 
into  account  the  needs  and  capabilities  of  the  web  developers  or  users.  Since 
implementation  of  a  standard  platform  could  directly  affect  the  workload  and  ease  of 
information  access,  it  is  important  to  consider  the  needs  of  the  primary  developers  and 
users  of  the  system  -  faculty,  administration  and  cadets  at  USAFA.  (Hazari,  1998). 


Although  a  particular  software  system  may  be  evaluated  on  the  basis  of  many 
technical  criteria,  in  the  end  the  system  must  ultimately  be  judged  by  its  usability:  To 
what  degree  does  it  help  users  accomplish  their  task(s)  and  to  what  degree  are  the  users 
satisfied  with  their  experience  (Levi  and  Conrad,  1996).  Therefore,  the  evaluation  of  the 
CPEG  interface  will  be  performed  at  two  levels,  the  functions  it  can  perform  and  the 
usability  of  those  functions.  These  will  be  tested  in  the  following  manner: 

a.  Requirements 

Commercial  off  the  shelf  navigational  interface  software  will  be  scored  using 
dichotomous  (yes/no)  variables  as  to  whether  it  meets  or  could  be  modified  to  meet  the 
requirements  as  outlined  in  Section  4. 1 .  This  initial  evaluation  will  be  conducted  by  the 
principal  investigator  and  validated  by  two  other  members  of  the  investigation  team. 

b.  Usability  testing 

A  group  of  USAFA  and  non-USAFA  study  subjects  (n  >10)  will  be  chosen  to 
perform  a  set  of  specific  usability  tests  on  the  interface.  Non-USAFA  personnel  will  give 
added  insight  to  impediments  that  may  be  experienced  by  users  new  to  USAFA  and  will 
consist  of  volunteers  from  other  academic  institutions  and  from  within  the  Department  of 
Defense.  In  addition  a  group  of  cadets  (n  >10)  will  also  be  chosen  to  perfonn  the  same 
set  of  evaluations.  The  tasks  they  will  be  asked  to  perfonn  consist  of  the  following: 

b.l  Layout  evaluation 

The  navigational  ability  of  CPEG  will  be  derived  in  part  by  the  intuitive  nature  of 
the  graphical  design  layout.  To  test  whether  this  layout  (irrespective  of  content)  helps 
users  find  key  elements,  all  text  on  a  page  can  be  replaced  with  unintelligible  nonsense, 
so  that  the  user  must  rely  on  the  intrinsic  communicative  aspects  of  the  layout  to  perform 
the  test  task  (Neilsen,  1998).  Therefore,  to  test  the  overall  layout  of  the  CPEG  design, 
each  test  subject  will  be  given  a  printout  where  the  words  have  been  replaced  with 
nonsense  (Figure  3).  They  will  then  be  asked  to  draw  blocks  around  key  elements  of  the 
page  such  as  main  content  selections  (courses,  and  other  links),  title,  search  feature,  last 
updated  date  and  time,  intranet  identifier  or  logo,  bulletins  and  news  items.  Users  will 
also  be  asked  to  detennine  which  elements  are  missing  from  the  page. 


Figure  3:  Layout  evaluation  of  the  CPEG  interface.  Figure  3a  shows  the  web  page 
as  it  would  be  normally  viewed,  whereas  Figure  3b  shows  the  same  interface  with 
the  text  replaced  by  nonsense. 


Figure  3a 


Figure  3b 

b. 2  Ease  of  use 

To  test  the  overall  usability  of  the  system,  the  system  will  be  set  up  using  web 
pages  developed  using  web  course  development  software  (Section  5.2c  below). 

Instructors  and  cadets  will  be  asked  to  log-in  to  the  system  and  perform  tasks  that  mimic 
real  use  scenarios.  For  example,  they  will  have  to  navigate  to  certain  locations,  modify 
the  calendar,  and  set  up  a  personalized  set  of  links.  Data  collected  will  include  the 
number  of  steps  in  the  process,  the  number  of  errors,  recovery  strategies,  time  on  task  and 
ease  of  use. 

c.  Heuristic  evaluation 

In  addition,  the  site  will  undergo  a  modified  heuristic  analysis  to  test  overall  web 
site  usability.  This  heuristic  evaluation  will  involve  a  set  of  experienced  web  design 
experts  (n  =  5)  who  look  for  violations  of  common  usability  principles  such  as  flexibility 
and  efficiency  of  use,  minimalist  design,  consistency,  and  user  error  prevention  (Instone, 
1999). 

5.2.  Evaluation  of  course  management  tools 

Since  course  management  features  will  be  an  integral  part  of  the  CPEG  interface, 
it  is  important  to  independently  test  the  capabilities  and  functionality  of  this  type  of 
software  to  ensure  that  the  overall  interface  meets  the  fundamental  needs  of  this  niche. 


Several  studies  have  evaluated  and  compared  various  aspects  of  course 
management  web  tools  (Gray,  1998;  Hazari,  1998;  InfoWorld,  1998;  Kristapiazzi,  1998; 
Marshall,  1998).  Of  the  products  available  commercially,  Blackboard’s  Courselnfo  was 
consistently  rated  among  the  best  course  management  software  due  to  its  variety  of 
features  and  ease  of  use  (Fredrickson,  1999).  However,  these  studies  generally  focused 
on  checklists  of  what  these  products  can  do,  not  what  these  products  need  to  do,  or  can 
do  well.  For  example,  packages  were  evaluated  by  whether  or  not  they  had  a 
collaborative  white  board  feature,  not  whether  the  whiteboard  feature  was  easy  to  use  or  a 
useful  component  of  the  package.  These  analyses  were  of  limited  predictive  power  in 
determining  the  applicability  of  different  software  because  two  or  more  products  may 
possess  the  same  feature  but  the  feature  may  not  function  in  the  same  manner.  Therefore, 
our  study  will  evaluate  the  usability  of  web  course  management  packages  following  the 
protocol  outlined  below: 

a.  Study  Participants 

A  group  of  instructors  (n  =  30)  and  cadets  (n  =  30)  from  USAFA  will  be  selected 
to  participate  on  a  volunteer  basis  to  evaluate  courseware  authoring  software  packages 
installed  on  the  computer  servers  at  the  Institute  for  Information  Technology 
Applications  (IITA)  in  Fairchild  Hall,  USAFA.  Participants  in  this  study  would  be 
matched  for  level  and  variety  of  experience  writing  HTML  from  raw  text,  using  HTML 
authoring  software,  and  programming  of  advanced  web  applications. 

b.  Requirements 

Software  packages  will  be  scored  using  dichotomous  (yes/no)  variables  as  to 
whether  they  have  certain  features  or  allow  for  the  development  of  key  course 
components  (Table  1).  The  three  software  packages  that  have  most  of  the  features  as 
outlined  in  Table  1,  and  that  come  closest  to  meeting  all  of  the  requirements  of  CPEG 
listed  in  Section  3.1,  will  undergo  usability  testing.  Faculty  members  will  also  rate  how 
important  they  feel  these  components  are  to  their  teaching  both  before  and  after  they 
learn  how  to  use  the  software.  If  appropriate  these  data  may  be  used  to  weight  the 
rankings  of  the  software  packages. 

c.  Usability  testing 

A  group  of  USAFA  study  subjects  will  be  chosen  to  perform  a  set  of  specific 
usability  tests  on,  and  develop  course  content  with  several  COTS  web  management 
software  packages.  In  addition,  a  group  of  cadets  will  be  chosen  to  perform  evaluations 
on  the  course  web  sites  produced  by  the  faculty  using  these  packages.  The  tasks  they  will 
be  asked  to  perform  consist  of  the  following: 

c.l  Layout  design 

Course  templates  are  an  integral  part  of  course  management  software  and  allow 
the  novice  web  developer  to  post  content  rapidly.  However,  before  a  template  is 
deployed,  it  would  be  advantageous  to  have  data  on  the  effectiveness  of  the  template 
itself.  This  is  often  difficult  to  do  because  the  layout  of  the  template  is  confounded  with 
its  content.  To  test  the  effectiveness  of  different  templates  they  will  be  tested  in  a  similar 
manner  as  described  above  in  section  5.1b.  Text  will  be  replaced  with  nonsense  and 


subjects  will  be  asked  to  draw  boxes  around  the  key  components  and  to  determine  which 
components  if  any  are  missing. 


c.2  Usability  of  course  authoring  system  by  instructors 

Instructor  study  participants  (n  =  30)  will  develop  a  web  site  with  a  syllabus, 
course  information  and  content  for  a  minimum  of  one  lesson  using  two  of  the  three  web 
course  management  tools  that  are  assigned  to  them  at  random.  Instructors  will  create 
course  content  using  the  web  authoring  feature  of  the  software  package  itself,  and  import 
pre-manufactured  pages  built  in  FrontPage,  PowerPoint,  text  built  HTML,  and 
Macromedia’s  Dreamweaver  to  determine  the  cross  compatibility  of  this  system  with 
other  development  tools  already  in  use  at  USAFA.  Data  will  be  collected  automatically 
and  with  surveys,  and  will  include  time  on  task,  ease  of  use,  and  a  comparative  rating  of 
similar  functions  for  the  two  course  management  tools. 

c.3  Usability  of  the  web  pages  by  cadets 

As  discussed  earlier,  the  development  of  course  materials  for  the  WWW  is  only 
part  of  the  puzzle.  Web  course  development  tools  must  be  evaluated  not  only  by  ease  of 
development,  but  the  usability  of  the  resulting  web  pages.  Few,  if  any  published  studies 
have  analyzed  this  component,  and  studies  performed  with  student  subjects  are  absent 
from  the  literature.  To  test  the  usability  of  the  course  web  sites  developed  with  the 
course  management  tools  in  a  realistic  scenario,  cadets  will  be  asked  to  analyze  the 
resulting  web  pages  that  are  developed  in  Section  5.2c.2  of  this  study.  For  each  course 
produced,  cadets  will  be  asked  to  access  the  course,  navigate  to  specific  areas  within  the 
course,  take  quizzes,  participate  in  discussions,  collaborate  on  group  assignments,  find 
their  grade,  and  perform  other  tasks  necessary  to  fully  test  the  functional  capability  of  the 
course.  Cadets  will  also  be  asked  to  make  predictions  on  what  they  will  find  when 
selecting  key  navigational  points  in  the  template.  Data  collected  will  consist  of  numbers 
of  steps  taken  to  reach  certain  navigational  goals,  ease  of  use,  perception  of  mental 
workload,  and  predictability  of  certain  functions. 

c.4  Usability  of  student  tracking  information  by  instructors 

Instructors  will  also  evaluate  the  student  tracking  features  of  the  system  by 
examining  real  data  from  the  cadets  that  log  onto  the  course  web-page,  as  well  as 
simulated  hits  from  the  investigators  if  needed.  Data  collected  will  be  similar  to  data 
collected  during  the  content  development  phase  (Section  5.2c.2)  of  this  study. 

6.  Laboratory  requirements 

This  project  will  require  a  computer  with  Microsoft  NT  operating  system  with  an 
ISS  server,  and  a  computer  with  a  Unix  based  operating  system  and  an  Apache  web 
server.  The  servers  must  be  accessible  from  outside  the  firewall  for  participation  by  non- 
USAFA  subjects.  Each  faculty  study  participant  will  need  a  password-protected  account 
with  ftp  and  telnet  privileges  that  allows  them  access  to  both  machines  for  development, 
and  storage  of  their  web  pages  in  the  CPEG  database.  Cadet  participants  will  only  need  a 
login  account  to  view  the  pages  and  submit  survey  data. 


7.  Investigation  team 

Principal  Investigator— Dr .  Margaret  E.  (Peg)  Halloran,  Director  of  Educational 
Technology,  IITA 

Human  Factors  Consultant-Dr.  James  Miller,  Human-Environmental  Research 
Center  (HERC),  United  States  Air  Force  Academy 
Laboratory  Configuration  Manager- Captain  Mance  Harmon,  IITA 
Instructional  Design  Consultant- Major  Stan  Supinksi,  Department  of  Foreign 
Languages 

8.  Proposed  timeline 

Oct  99  -  design  survey  and  metrics  for  data  collection 
Nov  99  -  download  and  install  course  management  software,  download  and 
install  demonstration  version  of  CPEG  development  software 
Nov/Dec  99  -  have  instructors  develop  web  pages,  and  collect  data.  Enter  web 
pages  into  the  CPEG  database 

Dec  99  -  review  web  pages  and  add  more  if  necessary  to  round  out  sample  for 
cadet  analysis.  Instructors  begin  evaluation  of  CPEG  interface 
Jan/Feb  00  -  have  cadets  evaluate  web  pages  and  CPEG  interface,  analyze 
instructor  data 

Mar  00  -  analyze  cadet  data,  write  up  preliminary  results,  begin  assessing 
hardware  specifications  and  implementation  plan 
Apr  00  -  present  preliminary  results  of  web  management  software  study 

9.  Product 

This  project  will  produce  a  report  that  includes  the  results  of  the  evaluation  and 
specific  recommendations  for  the  creation  of  a  CPEG  at  the  Air  Force  Academy.  This 
report  will  also  be  distributed  to  relevant  Air  Force  organizations 
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Table  1:  Criteria  for  Evaluation  of  Web  Course 

Development  Tools  for  Usability  Testing  Selection 
(Section  5.2b) 

Yes  No 

General  Features 

Static  toolbar 

Table  of  Contents 

Customized  "look  and  feel"  (colors,  icons,  logos,  etc) 

Automated  glossary  tool 

Automated  link  to  course  material  content 

Automated  indexing  tool 

Search  tool  for  course  material 

Student  can  make  private  annotations  of  course  material 

Integrated  Calendar  tool 

Multi  language  support 

Instructor  can  define  groups  of  students 

—Can  assign  specific  material  to  individual  or  group  of  students 

—Collaborative  work  area  for  group 

—Group  presentation  area 

Individual  presentation  area/homepage 

Student  file  upload  capability/instructor  comments 

Conforms  to  IMS  specifications 

Conforms  to  W3C  specifications 

Authoring  features 

Does  not  require  knowledge  of  HTML 

Allows  Java  applets 

Requires  plug-ins 

Has  drag  and  drop  authoring  features 

Allows  Macromedia  files 

Allows  FrontPage  files 

Allows  links  to  custom  developed  pages 

Allows  custom  graphics 

Course  Management  Features 

Class  list  can  be  entered  one  student  at  a  time,  or  uploaded  as  a  fde 
Class  list  can  be  uploaded  as  a  fde 

Class  lists  can  be  presented,  saved,  and  printed  using  Excel 

Online  student  manual 

Online  instructor  manual 

Course  can  be  downloaded  for  safekeeping 

Courses  can  easily  be  moved  from  one  server  to  another 

Directory  upload  capability  from  desktop 

Student  tracking  features 

—can  track  how  often  student  accesses  pages 

—can  track  when  student  accesses  pages 

—can  track  time  spent  on  each  page 


Communication  Features 

One  to  one  course  email 
One  to  many  course  email 
Searchable  asynchronous  discussion 
Logged  synchronous  discussion 
Virtual  field  trips  within  discussion  pages 
Shared  whiteboard 

Adaptable  for  desktop  video-teleconferencing 


Grading  Features 

Student  access  to  progress  data  available 
Ability  to  add  offline  grades 
Grade  statistics  and/or  histograms 
Instructor  comments  available  with  grade 
Scores  can  be  emailed  to  instructor 
Scores  can  be  stored  on  server 
Scores  can  be  exported  into  Excel 


Quizzing  features 

Quizzes  automatically  graded  and  entered  into  grade  book 

Allows  for  the  following  automatically  graded  question  types: 

—True  -False/Multiple  choice 

—Fill  in  the  blank 

—List  matching 

—Essay  questions 

—Imagemap  (click  on  correct  part  of  image) 

—Short  answer 

Questions  can  have  multiple  correct  answers 
Can  use  a  mixture  of  question  types  on  a  single  quiz 
One  question  at-a-time  testing  capability 
Question  fde  upload  capability 
Customized  feedback 

Redirect  path  dependent  on  question  answers 
Timed  quizzes 

Delivered  on-line  on  a  predetennined  time  and  day 
Supports  graphics  fdes  adjacent  to  quiz  question 
Supports  both  tutorial  and  real  exam  scenarios 
Random  assignment  of  questions  to  exams 
Allows  weighting  of  questions  so  students  get  equal  quizzes 


Can  be  used  with  the  following  server  /operating  systems 

Unix 

NT 

Macintosh 

Solaris 

Linux 

Other 


ABOUT  THE  INSTITUTE 


The  Institute  for  Information  Technology  Applications  (IITA)  was  formed  in  1998  to  provide  a  means  to 
research  and  investigate  new  applications  of  information  technology.  The  Institute  encourages  research  in 
education  and  applications  of  the  technology  to  Air  Force  problems  that  have  a  policy,  management,  or 
military  importance.  Research  grants  enhance  professional  development  of  researchers  by  providing 
opportunities  to  work  on  actual  problems  and  to  develop  a  professional  network. 

Sponsorship  for  the  Institute  is  provided  by  the  Secretary  of  the  Air  Force  for  Acquisition,  the  Air  Force 
Office  of  Scientific  Research,  and  the  Dean  of  Faculty  at  the  U.S.  Air  Force  Academy.  IITA  coordinates  a 
multidisciplinary  approach  to  research  that  incorporates  a  wide  variety  of  skills  with  cost-effective  methods 
to  achieve  significant  results.  Proposals  from  the  military  and  academic  communities  may  be  submitted  at 
any  time  since  awards  are  made  on  a  rolling  basis.  Researchers  have  access  to  a  highly  flexible  laboratory 
with  broad  bandwidth  and  diverse  computing  platforms. 

To  explore  multifaceted  topics,  the  Institute  hosts  single-theme  conferences  to  encourage  debate 
and  discussion  on  issues  facing  the  academic  and  military  components  of  the  nation.  More  narrowly 
focused  workshops  encourage  policy  discussion  and  potential  solutions.  IITA  distributes  conference 
proceedings  and  other  publications  nation-wide  to  those  interested  or  affected  by  the  subject  matter. 


Comments  pertaining  to  this  report  are  invited  and  should  be  directed  to: 
Sharon  Richardson 

Director  of  Conferences  and  Publications 

Institute  for  Information  Technology  Applications 

HQ  USAFA/DFPS 

2354  Fairchild  Drive,  Suite  6L16D 

USAF  Academy  CO  80840-6258 

Tel.  (719)  333-2746;  Fax  (719)  333-2945 

e-mail:  sharon.richardson@usafa.af.mil 


